PROGRAM EVALUATION
[INSTITUTION] Course Redesign Initiative
[DATE]

INTRODUCTION

Summarize the initiative’s history.
List the courses that were redesigned.
List the goals of the initiative.

PROCESS

Year One

Stage #1: Program Development

Describe activities.
Stage #1 was completed by [DATE].

Stage #2:  Building Awareness and Commitment

Describe activities—including number of participants—in Workshop I.
Stage #2 was completed by [DATE].

Stage #3: Orientation, Selection and Training of Course Redesign Teams

Describe activities—including number of teams responding to readiness instrument and number of teams and participants—in Workshop II.
Stage #3 was completed by [DATE].

Stage #4:  Individualized Consultation during Planning Phase

Describe activities, including number of teams submitting full proposals.
List those selected and those not selected to participate in the program.
Stage #4 was completed by [DATE].

Year Two

Stage #1: Preparing and Implementing Pilots

Describe activities, including number of pilots conducted.
Stage #1 was completed by [DATE].

Stage #2: Mid-Course Assessment

Describe activities—including number of teams participating in Workshop III—and state the number of teams dropped, if any, after the pilot implementation.
Stage #2 was completed by [DATE].

Year Three

Stage #1: Full Implementation

Describe activities—including number of teams participating—in Workshop IV.
Stage #1 was completed by [DATE].

Stage #2: Assessing the Impact

Describe activities, including number of completed projects.
Stage #2 was completed by [DATE].

THE RESULTS

Summarize the course redesign projects’ final reports.
There were [#] completed course redesign projects.

1) Did student learning improve (as measured by direct comparisons of content mastery)?

[#]        Yes
[#]        No difference
[#]        No

2) Did course completion rates improve (as measured by comparisons of final grades)?

[#]        Yes
[#]        No difference (but traditional completion rates were high)
[#]        No difference
[#]        No (but standards were higher in the redesign)
[#]        No

3) Were instructional costs reduced?

[#]        Yes
[#]        No

4) Will the redesign be sustained after the grant period is over?

[#]        Yes
[#]        Unclear
[#]        No

1) Did student learning improve (as measured by direct comparisons of content mastery)?

Yes

1. Basic English (used pre- and posttests to compare student writing samples by using a common rubric and compared grammar and mechanics by using a common online diagnostic)

• It appears that there was some improvement in student learning. Two sets of pre- and posttest scores were collected from students: one from a rubric designed in-house and the other from an online diagnostic designed by Pearson publishing.

• Students in the traditional course showed a 14-point gain on their rubric finals, whereas students in the redesigned course showed a gain of 17 points. Students in the traditional course showed an 18-point gain on the online diagnostic, whereas students in the redesigned course showed a 5-point gain. It is worth noting that the in-house rubric had a grammar and mechanics portion, and redesign students outscored the traditional ones.

2. General Chemistry (compared common final exams)

• There was a considerable increase in performance on the postassessment from the fall 2011 traditional course (mean=70.93%) to the fall 2012 redesigned course (mean=80.39%).

• In addition, a math and science preparedness test developed by the team was administered to all students as a pretest. Students in the redesign course were less prepared (mean score=68.56%) than students in the traditional course (mean score=74.70%), making the learning increase in the redesigned course even more impressive.

3. Introductory Psychology (compared common final exams)

• The fall 2011 semester served as a baseline semester wherein two faculty members, both of whom were members of the redesign team, each taught a traditional section of Introductory Psychology. The primary measures of learning outcomes included two (pre- and post-) comprehensive exams: (1) a 30-item comprehensive exam that had been developed a number of years ago by members of the department of psychology and that has traditionally been used as a measure of student learning in Introductory Psychology and (2) a 50-item comprehensive exam created specifically for this project by the course redesign team.

• Analyses confirmed that there were no significant differences between the two baseline sections with respect to pretest scores on the 30-item exam (t (276) = .92, p > .05) or the 50-item exam: t (282) = .04, p > .05). Thus the sections were combined into one comparison group (n = 302).

• The full implementation occurred during the fall 2012 semester and consisted of five sections, all taught by members of the redesign team. Results indicated that on the 30-item comprehensive exam, students (n = 1340) in the redesigned sections performed significantly better (84% improvement) compared with the traditional comparison group (54% improvement): t (317.54) = -7.50, p < .001. Similarly, students in the redesigned course demonstrated significantly more improvement from pre- to posttest on the 50-item comprehensive exam (62% improvement) compared to the traditional sections (37% improvement), t (429.41) = -12.55, p < .001.

4. Intermediate Algebra (compared common final exams)

• Students in the redesigned course performed significantly better. With a baseline of 100 points, the average final exam score for the fall 2011 traditional sections was 63; that for the fall 2012 redesigned sections was 85.

5. College Algebra (compared common final exams)

• Students in the fully implemented redesigned course performed significantly better on the final exam compared with students in the previous three fall semesters of the traditional course. The mean on the final exam in the redesigned sections was 68%; the mean in the traditional sections was 55%.

6. Computers and Information Systems (compared common final exams)

• In the traditional course, final exam scores averaged 73%, whereas in the redesigned course, final exam scores averaged 79%.

• Pell-eligible students' average final exam scores was 69% in the traditional course and 77% in the redesigned course. 

No difference

1. Introduction to Business (compared common final exams)

• A common comprehensive final exam measured academic performance between the traditional and redesigned courses. Students in the traditional course scored an average of 74%; students in the redesigned course scored an average of 73%.

No

1. Oral Communications (compared common final exams)

• Students in the redesigned section of the course had a lower average score on the final assessment evaluated student performance. The average score for traditional sections was 71.3%; the average score for the redesigned sections was 66.6%.

2) Did course completion rates improve (measured by comparing final grades)?

Yes

1. Intermediate Algebra

• Student success rates (final grade of C or better) increased from 68% in the traditional sections to 85% in the redesigned sections.

• The percentage of students receiving an A in the redesigned sections was more than twice the percentage of those in the traditional section (48% vs. 23%). 

• It is worth noting that some instructors of the traditional sections curved the grades of the sections they taught, whereas all of the redesigned sections were graded on a straight scale.

2. College Algebra

• Course completion (with final grades of C or better) increased slightly in the redesign from 63.8% in three previous fall semesters to 66.5% in the redesigned semester.

No difference

1. General Chemistry

• There was no significant difference in the completion rate in the redesigned course, with 77.23% of students receiving C grades or better in the redesign compared with 78.30% in the traditional course.

2. Introductory Psychology

• Completion rates (grades of C or better) were 76% in both the traditional and redesign sections.

3. Computers and Information Systems

• There was no significant difference in the final grades for the course: 85% of students in the traditional course received passing grades (defined as C or better) versus 84% of students in the redesigned course.

No, but standards were higher in the redesign

1. Basic English

• The percentage of students earning grades of C or better in the traditional course was 50%; in the redesigned course, it was 36%.

• Because grade inflation was an issue in the past affecting approximately 20% of students, the lower number mastering the course in the redesign is not alarming. In addition, exit standards were not only consistent in the redesigned course but also higher than in the traditional course.

2. Oral Communications

• The percentage of students earning grades of C or better in the traditional course was 65%; in the redesigned course, it was 64%.

• However, final grades from the traditional course suffered from grade inflation. Even though the course had a syllabus as a guideline, the preponderance of the 18 adjuncts instructing sections viewed the guideline in differing ways. Some of the sections did not require speeches; rather, those sections replaced actual speeches with rhetorical criticisms of speeches. Those sections tended to have curved grades that were mostly A or B. In the redesign, the undergraduate learning assistants were trained to follow National Communication Association guidelines for oral communication presentations to standardize what an A or B speech should be. Most adjunct instructors in the traditional course could not have identified those competencies in an oral presentation.

No

1. Introduction to Business

• The percentage of students earning grades of C or better in the traditional course was 69%; in the redesigned course, it was 60%.

3) Were instructional costs reduced?

Yes – Saved More

1. General Chemistry

• The cost savings plan was to reduce the number of instructors needed to teach the course by increasing the section size from 200 to 400 students, reducing the number of sections offered annually from six to three, and transferring some student learning experiences online. Those actions would reduce the cost per student by 19%, from $150 to $122.

• The team carried out its cost savings plan of reducing the number of instructors from six to three (two in the fall and one in the spring, down from four in the fall and two in the spring) and combining original sections into larger sections of students that included face-to-face and synchronous online participation.

• In addition, the team saved more than it had anticipated, because the redesign required fewer graduate teaching assistants (GTAs) than originally planned. Only 7 GTAs were needed instead of the 12 projected. The structure of the activities enabled undergraduate learning assistants (ULAs) to replace GTAs. Three additional ULAs were hired, but at a lower rate than GTAs.

• Furthermore, with only three sections taught annually and a much more organized course, released time for coordination was not needed in the redesign. The actual cost per student dropped from $150 to $102 per student, a savings of 32%.

2. Oral Communications

• The cost savings plan was to decrease the number of sections offered annually from 44 to 4, to increase section size from about 23 to 230 students, and to decrease the need for adjunct instructors from 18 to 2. Four undergraduate learning assistants would work with small groups of students both in the lecture and in the Communications Assessment and Learning Lab. The cost per student was projected to decrease from $174 in the traditional format to $39 in the redesigned format.

• The team saved more than originally projected. Because enrollment dropped from 920 students to 755, the team decreased the number of sections offered annually from 44 to 3. The cost per student decreased from $174 in the traditional format to $33 in the redesigned format, a reduction of 81%.

3. Intermediate Algebra

• The plan was to reduce the number of Intermediate Algebra instructors, all of them graduate teaching assistants, from 5 to 2.5 by increasing section size from 35 to 70 students and thus reducing the number of sections offered annually from 20 to 10. Although undergraduate learning assistants were added to help staff the computer lab, the redesign would reduce the cost per student from $118 to $104, a 12% savings.

• During full implementation in the fall 2012 semester, the redesign team was surprised by a high enrollment of 497 students in Intermediate Algebra (compared with the estimated average of 375 students). Therefore, eight sections of Intermediate Algebra, instead of five, were offered taught by three GTAs. Despite that modification to the plan, the cost per student reduced from $118 to $103 per student, a 13% savings.

4. Computers and Information Systems

• The original plan was to consolidate all sections under one instructor assisted by a team of undergraduate and graduate students. The number of sections would decrease from 11 to 2 annually, and section size would increase from 50 to about 150 to 300 students. The planned enrollment was 550 students and would have resulted in a decrease in cost per student from $113 to $95, a 16% reduction.

• However, the actual enrollment increased to 640, yielding greater savings.  The cost per student decreased from $113 in the traditional sections to $81 in the redesign, a 28% reduction.

Yes

1. College Algebra

• Costs were reduced as planned by more than doubling student load for GTAs and adjunct instructors: from one section of 35 to 40 students to two sections of 50. Removing GTAs and adjunct instructors from the lecture setting and placing them in the lab setting means they no longer spend time preparing lectures. Additionally, far less time is spent grading student work. Those changes reduced the cost per student from $103 to $67, a 35% decrease. Instructors found, however, that they had somewhat underestimated the time needed for interacting with students outside of scheduled class meetings, for proctoring exams, for grading, and for training.

Yes – Saved Less

1. Introductory Psychology

• The cost savings plan was implemented as originally planned and consisted of a combination of the restructuring of course personnel and increased section size. The course staff changed from one faculty member or adjunct instructor teaching 153 students to a teaching team of seven individuals (one full-time faculty, one graduate student or adjunct instructor serving as senior learning assistants, and five undergraduate learning assistants [ULAs]) for a section of 300 students. The anticipated reduction in cost per student was 18%.

• The number of sections taught annually declined from 18 to 9 as planned. Three changes affected the actual cost per student: (1) The team expected a small enrollment increase of 72 students annually, which did not occur. Instead, enrollment declined by about 126 students to 2,500. (2) Two of the sections were taught by an experienced adjunct rather than by a full-time faculty member. (3) The team decided the course needs a coordinator to ensure consistency and to train multiple course staff as staff change in the future. The result of those changes was that cost per student actually decreased from $73 in the traditional course to $66 in the redesign, a 10% reduction.

2. Introduction to Business

• The team carried out its cost reduction plan of using fewer full-time faculty to teach the course. In the traditional format, typically four faculty members taught the course in the fall, and three in the spring. In the redesign, the plan was for one full-time faculty member to teach the course to one large section of students (150 to 160 in fall and 75 to 100 in spring) with support from adjuncts and ULAs.

• Cost reduction was, however, less than anticipated. In the past, typical annual enrollment had been 200. During the course of the redesign, the university dropped its Associate in Business degree, which had required that students take the Introduction to Business course. That change unexpectedly dropped the annual enrollment by nearly half. With the reduction in sections, instruction was carried out by one faculty member and one adjunct. The cost per student was projected to decline by 60%, from $325 to $130. Instead, it declined by 33%, from $325 to $217.

No

4) Will the redesign be sustained after the grant period is over?

Yes

1. Basic English

• The redesign should be sustainable. One issue that needs to be addressed is finding high-quality undergraduates—with schedules that fit around the course’s schedule—to serve as ULAs. In the future, assistants will be interviewed and secured months in advance.

2. General Chemistry

• The newly hired faculty position is permanent; therefore, there is support for the redesigned format of the course to continue.

• There are plans to increase students’ number of options, including an online asynchronous model. Additional, discipline-specific recitation models are also being planned.

3. Introductory Psychology

• Support for the redesigned course has not wavered within the department and the university, and there is every indication that the course will continue as currently designed. All of the members of the original redesign team had their teaching responsibilities modified to accommodate the course redesign process, and now those faculty wish to return to teaching other courses in addition to introductory psychology. That change will require the recruitment of additional department faculty interested in teaching the redesigned course. (This has already begun, with a combination of existing faculty and new hires.)

• Overall, the redesign team developed a robust, engaging course, based on best practices in teaching that have proved to significantly improve learning outcomes. Therefore, any revisions that take place in the future are likely to be relatively minor and are not expected to place any significant burden on the existing redesign team.

4. Intermediate Algebra

• The full implementation of redesigned Intermediate Algebra in fall 2012 occurred just in time, when more math full-time faculty and adjuncts were needed to teach other courses due to increased enrollment and expanded programs within the department. Meanwhile, a lot more students than expected were required to take Intermediate Algebra. The department had to offer eight 70-seat sections instead of five sections as originally planned. Without the redesign approach, it would have been impossible to offer three more sections accommodating nearly 200 additional students on short notice.

• The higher course completion rate makes it clear that the redesign approach improves the student learning experience. The two redesigned College Algebra sections were full shortly after they opened for enrollment, which shows that many students were in favor of the redesign model when it was available. At the same time, the Department of Academic Enrichment is seeking to undertake Introductory Algebra redesign with the help of the Intermediate Algebra redesign team.

5. College Algebra

• The redesign has shown it is a cost-effective way to improve student learning and reduce course drift.  No department faculty are interested in moving back to the traditional format, and the administration remains committed to the redesign effort. Students are reasonably content with the format: in the full implementation there were no complaints to the department chair.  The redesign will be sustained.

Unclear

1. Oral Communications

• Student outcomes showed no significant difference in learning, whereas instructional costs reduced by 81%. However, if the course does not receive public support from campus leaders along with the required technology and data assessment needed to win success, the redesign has a clear opportunity to backslide. The cost savings and format of the course are sustainable as long as the selection and training of learning assistants remain priorities.

2. Introduction to Business

• The team plans to continue the redesign for at least two more years because the enrollment numbers will stay high enough for at least that length of time. The course redesign was certainly worth the time and effort. Two other disciplines on campus have already adopted redesign models for their large course sections, and many others are considering such adoption. This will result in a net gain for the university as a whole.

3. Computers and Information Systems

• Despite the best efforts of the redesign team and lead instructor to communicate the goals and structure of the redesign, institutional changes in leadership have posed a number of obstacles. The department chair left just before the full implementation, and the dean of the college of business administration is stepping down. As the information systems department shrinks in faculty full-time equivalents and students and as the institution faces substantial budget cuts in the near future, the challenge is to lobby for the necessary resources: a large teaching space to accommodate all students at one time.

No

Edited copies of final reports from each completed redesign are in Attachment A. Final reports include learning-outcomes data, course completion data, cost reduction data, a discussion of the most-important pedagogical techniques that led to increased learning, a discussion of the most-important cost reduction techniques that led to reduced costs, a discussion of implementation issues encountered during the redesign process, and a discussion of future sustainability of the redesign.

OBSERVATIONS

Following are examples of observations that NCAT has made.

• There are significant interest in and understanding of the value of course redesign on campus. Most faculty, staff, and administrators indicated that once they understood the interrelationship between cost and quality and learned about strategies to address both simultaneously, they became willing to redesign. Redesign project participants said they now understand clearly that it is possible to reduce costs while simultaneously increasing or maintaining quality as a result of participating in the initiative. In several cases, departments and institutions are moving to adapt the redesign methodology to other courses beyond the first course that was redesigned.

• The workshops and consulting sessions helped participants understand NCAT’s strategies for quality enhancement and cost reduction. Once the teams participated in the workshops, they were much more prepared to formulate strategies for both quality enhancement and cost reduction. That greater preparedness was evident in the proposals the teams submitted. Prior to their experience in the planning process, it was difficult for some faculty to imagine how to approach the issues simply by referring to NCAT’s website. The workshops were key in providing examples, presenting the organizing principles, conveying a national perspective, and offering opportunities for discussion. Unfortunately, in three cases, the final project leaders inherited the redesign from the original project leaders once the project was under way and did not participate in the workshops.

• Project teams understood the amount of work needed for the redesign and strategies for approaching the issues. Most teams were quite well organized and allocated the design work among members. Those teams moved expeditiously through the planning and development process by spending their time effectively, and they achieved success and sustainability. Projects that did not form project teams tended to struggle and be less successful.

• All projects focused on courses with multiple sections. Courses with large numbers of students and multiple sections provided the opportunity, in most cases, to show significant cost savings—on average, 30%.

• Faculty were not as familiar with quantitative evaluation strategies that allow for comparison between traditional and redesigned formats and that demonstrate improved student learning as a result of redesign efforts. This is consistent with observations of initial faculty confusion in other NCAT course redesign programs. However, with some assistance, faculty became able to design assessment plans for establishing baseline data and comparing learning in the traditional and redesigned courses. During the workshop and consulting sessions, teams found discussion of possible strategies very useful, and all were able to identify methods they could use to implement a successful assessment plan.

• Faculty were not at all familiar with costing strategies that facilitate comparison between traditional and redesigned formats and that document reduced instructional cost as a result of redesign efforts. This is consistent with observations of initial faculty inexperience in other NCAT course redesign programs. However, with a great deal of assistance, faculty became able to develop cost reduction plans that established baseline data and able to compare costs in the traditional and redesigned courses. During the workshop and consulting sessions, teams found discussion of possible strategies very useful, and all were able to identify methods they could use to implement a successful cost reduction plan.

• Active intervention and frequent monitoring by NCAT staff during the project implementation period greatly increased the likelihood of success. Based on NCAT’s experience with many large-scale course redesign projects, we know that projects can get derailed during the implementation phase because of unanticipated issues that arise. Teams may not know how to respond, and their initial reaction is often to revert to the status quo of the traditional model. At those junctures, teams that sought advice benefited from experts in course redesign who could discuss the problems and offer strategies to resolve them. Other teams soldiered on without the benefit of that advice, often to their own detriment.

• A diversity of disciplines was involved and successful. The redesign projects included the natural sciences (human anatomy and chemistry); two in mathematics, two in English/communications, and one in statistics; and one each in psychology, computing, business, management, and health, thus demonstrating that course redesign principles can be implemented successfully in a variety of disciplines.

• The funded projects produced a number of models. [INSTITUTION] now has a number of models for future redesign initiatives within its colleges and universities. These provide a range of approaches complementing those already available at NCAT.

• A number of projects demonstrated increased student-learning outcomes or similar student-learning outcomes but reduced completion rates. Several of the projects investigated that contradiction and discovered that prior grade inflation was most likely the cause. Other projects experiencing the same phenomenon should also investigate whether prior grade inflation played a role. [INSTITUTION] should conduct a review of grading practices in view of possible inconsistencies between student-learning outcomes and grades awarded. When there is a contradiction between increased student-learning outcomes and decreased completion rates, we believe, as do the project leaders, that in most cases it’s a result of prior grade inflation or prior inconsistency in grading standards.

• Despite mixed results in improving learning and course completion, the vast majority of project leaders are fully supportive of the redesigns. For a variety of reasons—as captured in the Sustainability section of this report and in the individual project reports—the majority of project leaders are strongly supportive of the continuation of their redesigns. NCAT is confident that learning outcomes will improve as the various designs address the issues that arose in the initial implementations. In the professional judgment of the faculty, the redesigns are effective in improving the quality of the course while at the same time reducing costs. Greater consistency of content and coverage, valid and reliable measurements of student learning, and greater student engagement in course content all serve to back up that judgment.

• Despite mixed results in improving learning and completion rates, all projects were able to reduce their costs. One of the powerful messages of course redesign is that achieving the goal of reduced cost can have a significant impact on institutions’ ability to deal with budget crises, serve more students with the same resources, and free faculty to do other institutional tasks—all with no diminution in quality.

RECOMMENDATIONS

Following are examples of recommendations that NCAT has made.

• [INSTITUTION] should find ways to publicize the results of the Course Redesign Initiative throughout its campuses.

• [INSTITUTION] should conduct a second round of the Course Redesign Initiative.

• [INSTITUTION] should create a [INSTITUTION] Redesign Scholars Program based on those who have both improved student learning and reduced instructional costs.

•  [INSTITUTION] should consider requiring course redesign as part of its campus allocation strategy.

Table of Contents

Introduction
I. The Critical Components of a Successful Course Redesign Program
II. Assessing Institutional Readiness to Redesign
III. Making Key Decisions before the Program's Launch
IV. Developing a Plan of Work
V. Building Awareness and Capacity
VI. Assessing Course Readiness
VII. Preparing Teams to Submit Strong Proposals
VIII. Selecting Proposals That Will Succeed
IX. Monitoring the Redesign Implementations
X. Maintaining Consensus and Ensuring Sustainability
XI. Building Capacity and Scaling Initial Success

Appendices:

A. Plan of Work
B. Publicity Plan
C. Call to Participate
D. Application Guidelines
E. Workshop I Agenda
F. Workshop II Homework
G. Workshop Logistics
H. Workshop II Agenda
I. Workshop III Invitation
J. Workshop III Agenda
K. Workshop IV Invitation
L. Workshop IV Agenda
M. Final Report Format
N. Program Evaluation